Image-adaptive Color Super-resolution

Image super-resolution is the problem of recovering a high resolution (hi-res) image from multiple low resolution (lo-res) acquisitions of a scene. The main focus and the most significant contributions of research in this area have been on the problem of super-resolving single channel (grayscale) images. Multi-channel (color) image super-resolution is often treated as an extension to grayscale super-resolution by simply considering the luminance component of the image more carefully than the chrominance components. In this paper we address explicitly the problem of color image super-resolution by formulating an optimization problem that leads to convergence guarantees. The key contribution of this work is the inclusion of a color regularizer that effectively accounts for both luminance and chrominance geometry in images. We show results demonstrating substantial image quality improvement over the state of the art, especially for images with significant chrominance geometry. Introduction The resolution of an imager is limited by the resolution of its image sensor and the quality of its optics. In several imaging applications it is useful to have the ability to recover an image with resolution higher than that permitted by the capabilities of the imager. Image super-resolution fills this need by recovering a high resolution (hi-res) image from multiple low resolution (lo-res) acquisitions of a scene provided, of course, that the different low-res images capture different (at sub-pixel level) views of the scene. There are several imaging applications where super-resolution finds use for instance, medical imaging applications that use images for computer vision tasks benefit from high-res images. Another application where super-resolution is particularly appropriate is in surveillance applications where a video stream, which can provide input lo-res frames to the super-resolution algorithm, is continuously acquired. In simple terms, the super-resolution problem is addressed by describing first the several lo-res images on a grid finer than the resolution of single images (an image registration problem) followed by filling in values for missing pixels (an image interpolation problem). There has been significant advance in super-resolution research in recent years. Park et al. [1] give an overview of the problem and describe early advances. Most performance improvements come from solutions to the image registration problem with better motion estimation techniques. A common thread in most work is the focus on grayscale image super-resolution. Color image super-resolution is often treated simply by assuming that the luminance component of the image carries its spatial features. Algorithms that consider the chrominance components will only use them to improve image registration by better motion estimation [2, 3]. Very few researchers consider explicitly the relationship between the color channels in the interpolation problem. When they do, a common approach is to assume that spatial high-frequency components across the color channels are strongly correlated. In other words, if an edge (or a feature) is sensed in one channel, it implies that the edge (or feature) exists in all channels. Farsiu et al. [4] use this approach in a joint demosaicking and super-resolution problem formulation with good results. We note that the assumption about strong interchannel correlation in high frequency components is akin to assuming that most spatial features (edges and texture) appear in some luminance-type component found either with decomposition to a standard luminance-chrominance space like YCbCr, or with the PCA technique for decorrelating the color components. This assumption is clearly untrue for images with strong chrominance geometry images in which edges and textures are not a result of ambient illumination but due to edges between objects with different chrominance. In this work we consider explicitly the problem of color image super-resolution. The key contribution of this work is the inclusion of a color regularizer that effectively accounts for both luminance and chrominance geometry in images. We propose an optimization framework that is separably convex, leading to convergence guarantees, along with the enforcement of constraints consistent with real-world imaging physics. We show results demonstrating substantial image quality improvement over the state of the art, especially for images with significant chrominance edge features. Image-adaptive color super-resolution We first present the mathematical formulation of our color SR framework. We use the camera imaging model [1]: yk =DBTkx+nk, 1≤ k≤ K , (1) where x= [xr x T g x T b ] T ∈ 3n is the unknown (vectorized) hires image that we seek to reconstruct (subscripts r, g, b indicate the red, green and blue color channels respectively), yk ∈ m represents the k-th observed lo-res image, Tk ∈ 3n×3n is the k-th geometric warping matrix, B ∈ 3n×3n describes camera optical blur, downsampling matrix D∈ m×3n models the aliasing, and nk ∈ m is the noise vector that corrupts yk. Single-channel super-resolution The standard SR reconstruction problem recovers an estimate of x by minimizing the error between the warped, blurred and downsampled versions of x as predicted by the imaging